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Emphasis  is  often  placed  on  the  sample  rate  or  spacing  of 
sampled  imagery,  with  less  attention  paid  to  how  the  output  is 
constructed  using  the  samples.  The  fidelity  of  a  sampling  process, 
however,  depends  on  the  reconstruction  method  as  well  as  on  the 
sainple  rate.  This  paper  will  discuss  the  benefits  of  display  processing 
to  improve  image  quality  in  sampled  sensors.  Topics  will  include: 
basic  sampling  theory,  the  application  of  Fourier  Transforms  to 
sampled  imaging  systems,  the  origin  of  phase  artifacts  in  sampled 
imagery,  and  a  discussion  of  the  benefits  of  display  processing  when 
using  more  display  pixels  than  sensor  samples.  Examples  will  be 
shown  of  images  generated  using  different  reconstruction  techniques. 


1.0  Introduction 


It  is  widely  recognized  that  a  small  sample  interval  is  important  in  order  to 
achieve  good  replication  of  sampled  imagery.  However,  the  fidelity  of  a  sampling 
process  is  also  affected  by  the  method  used  to  construct  the  output.  In  many  applications 
one  display  pixel  is  used  for  each  sensor  sample.  Further,  newer  systems  might  use  flat 
panel  displays  with  square,  rectangular  or  round  pixels  which  are  unlikely  to  match  the 
shape  of  the  original  image  between  the  samples.  Depending  on  display  resolution,  this 
mismatch  might  result  in  poor  imagery  or  image  artifacts. 

A  higher  resolution  display  is  required  to  improve  the  displayed  image,  but  it 
may  not  be  necessary  to  take  additional  sensor  samples.  Once  a  minimum  sample  rate 
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has  been  reached  for  a  given  sensor,  improved  display  techniques  can  substitute  for 
increased  sampling  and  perhaps  save  system  cost  and  complexity.  The  shape  of  the  signal 
between  the  samples  can  be  estimated  or  "interpolated"  from  the  sample  data;  these 
interpolated  values  can  then  be  used  to  make  the  displayed  image  a  closer  match  to  the 
original  scene. 

The  benefits  of  display  processing  to  improve  image  quality  will  be  discussed. 
In  Section  2.0,  the  process  of  sampling  and  reconstructing  a  signal  is  illustrated;  the 
dependence  of  replicated  signal  fidelity  on  reconstruction  technique  is  shown.  In  the 
next  section,  two  expressions  for  the  Fourier  Transform  of  the  reconstructed  signal  are 
derived.  The  first  expression  is  used  to  clarify  the  reasons  why  a  "system  function"  can 
not  generally  be  associated  with  a  sampled  process.  The  second  Fourier  expression 
provides  more  insight  into  the  signal  reconstruction  process  and  is  the  basis  for  most  of 
what  follows. 

Section  4.0  of  the  paper  provides  an  example  of  typical  display  processing  and 
describes  the  associated  problems:  loss  of  high  frequency  components  in  the  signal  and 
sample  phase  artifacts.  Alternate  reconstruction  methods  are  then  discussed  in  Section 
5.0  and  examples  given  in  Section  6.0  to  illustrate  the  benefits. 


2.0  Sampling  and  Reconstruction 


In  Figures  1  through  3,  the  function  f(x)  is  sampled  by  finding  the  value  at 
uniformly  spaced  intervals  as  indicated  by  the  large  asterisks  (*).  If  (n)  samples  are 
taken  with  spacing  (X),  an  approximation  g(x)  to  f(x)  can  be  reconstructed: 


g(x)  =  I  f(nX)T(x-nX)  Equation  1 

alln 

where  r(x)  is  a  reconstruction  function.  The  figures  show  three  versions  of  g(x)  using 
different  r(x);  in  each  case,  r(x)  is  shown  graphically  as  an  insert.  Figure  1  uses  a 
rectangular  function  for  reconstruction;  bi-linear  interpolation  is  used  in  Figure  2.  In 
Figure  3,  r(x)  is  the  product  of  a  sine  function  times  a  Gaussian  function.  The  selection 
of  reconstruction  function  r(x)  and  sample  interval  (X)  are  fundamental  to  the  fidelity 
with  which  g(x)  approximates  f(x).  In  each  of  the  examples  shown,  g(x)  equals  f(x)  at  the 
sample  points;  the  match  between  g(x)  and  f(x)  improves  from  Figure  1  through  Figure  3 
because  the  different  r(x)  provide  progressively  better  interpolation  between  sample 
points. 


3.0  Fourier  Transform  of  the  Reconstructed  Signal 


This  section  provides  background  for  the  subsequent  discussion  on  how  the 
display  affects  image  quality.  The  concept  of  reconstruction  filter  is  described,  and  the 
reasons  given  why  a  Modulation  Transfer  Function  (MTF)  can  not  be  defined  for 
sampled  systems. 

If  R^)  is  the  Fourier  Transform  of  r(x)  defined  in  Section  1.0,  then 
R(w)-exp-j''^  is  the  transform  of  r(x-nX).  Since  the  reconstructed  signal  is  just  a  sum 
of  offset  reconstruction  pulses  r(x-nX),  the  Fourier  Transform  of  the  reconstructed 
signal  g(x)  in  Equation  1  is 


G(w)  f(nX)-R(w)-exp-jwnX 


Equation  2 


Equation  2  helps  to  illustrate  why  a  MTF  or  "system  function"  can  not  be 
assigned  to  a  sampled  process.  An  MTF  exists  if  the  input  to  the  system  can  be  shifted 
without  changing  the  frequency  content  of  the  output;  the  output  would  shift  coincident 
with  the  input,  but  the  shape  would  not  be  altered.  If  the  output  exhibits  this  invariance 
under  a  shift  of  the  input,  the  system  is  said  to  be  constant  parameter,  and  the  output 
spectrum  will  be  the  product  of  the  input  spectrum  and  a  system  function. 

In  Equation  2,  R(w)  is  not  acting  to  weight  or  filter  the  input  spectrum  F(w); 
that  is,  G(w)  does  not  equal  F(w)-R(w).  In  the  limit  of  (X)  small  and  (n)  large,  the 
discrete  transform: 

Fdiscrete(w)  =  X 

does  approximate  the  continuous  transform  F(w)d  however,  in  most  practical  systems, 
the  sample  rate  is  limited  and  the  attributes  of  a  constant  parameter  system  will  be 
missing.  That  is,  a  sinusoidal  input  at  frequency  wq  will  not  necessarily  and  solely  result 
in  a  sinusoidal  output  at  the  same  frequency  with  amplitude  ratio  R(wq).  Further,  the 
output  spectrum  G(w)  is  not  uniquely  determined  by  F(w)  and  R(w),  but  also  depends 
on  how  (where)  f(x)  is  sampled.^ 

An  alternate  and  potentially  more  useful  expre.ssion  for  the  Fourier  Transform 
of  g(x)  can  be  derived.  By  treating  the  output  as  having  resulted  from  convolving  r(x) 
with  delta  functions  at  the  sample  points,  Equation  1  can  be  re-written:^ 
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=  I  {f(x)-5(x-nX)}*r(x) 
all  n 


where  (*)  represents  convolution.  Since  convolution  in  space  results  in  multiplication  in 
frequency,  G(w)  becomes: 

Cf(w)  =  Y.  {F(w)  e‘j'^^ (w-nwc)}-R(w) 
alln 

Wg  =  l/(27tX) 

and  r(x)  is  assumed  to  be  symmetrical  about  zero  (a  sample  point).  For  generality  the 
zero  of  f(x)  can  offset  by  a  distance  (d).  Then: 

G(w)  =  y  F(w-nw<.)-e‘j(''^‘^'''s)^-R(w)  Equation  3 

alln 

Figure  4  is  a  notional  plot  of  G(w).  The  output  spectrum  can  be  represented  as  F(w) 
replicated  at  multiples  of  Wg  and  then  weighted  by  R(w).  Note  that  the  figure  just  shows 
the  amplitude  of  the  various  spectra;  the  phase  of  the  F(w)  replicas  varies  due  to  the 
eJnWgd  multiplier.  The  shape  of  the  output  waveform  (i.e.,  the  inverse  transform  of  the 
"spectrum  of  reconstructed  signal"  shown  in  the  figure)  will  vary  with  sample  phase. 

Equation  3  shows  why  R(w)  is  sometimes  called  a  "reconstruction  filter."  In  this 
expression  for  G(w),  R(w)  is  a  multiplicative  factor  and  has  the  appearance  of  a  system 
function.  This  expression  for  the  transform  shows  clearly  what  the  desirable  properties 
of  R(w)  should  be:  unit  amplitude  response  in  the  frequency  band  of  the  original  signal 
and  a  sharp  drop  in  response  at  higher  frequencies.  Equation  3  also  provides  insight  on 
the  implications  of  using  a  less  than  ideal  reconstruction  function. 

It  should  be  noted  that  the  sum  of  F(w-nwQ)  terms  does  not  represent  the 
frequency  spectrum  of  a  "sampled  f(x)"  but  rather  the  transform  of  a  series  of  delta 
functions.  The  delta  functions  do  not  have  physical  meaning  outside  their  defining 
integral, but  we  often  engage  in  the  convenient  engineering  fiction  that  the  spectrum 
consisting  of  all  F(w-nwQ)  terms  is  the  transform  of  a  series  of  infinitely  bright, 
infinitesimally  small  sp>ots  at  the  sample  points.  It  is  these  very  small,  very  bright  spots 
that  we  are  "filtering"  with  the  reconstruction.  The  actual  samples  are  just  the  value  of 
f(x)  at  spaced  intervals;  it  Ls  not  physically  sensible  to  associate  a  Fourier  Transform  with 
a  list  of  numbers. 


In  order  to  achieve  a  good  reproduction  of  the  sampled  signal,  the  sampling 
interval  (X)  must  be  small  enough  that  the  replicas  of  F(w)  do  not  overlap  significantly; 
that  is,  aliasing  must  be  minimized.  The  reconstruction  function  r(x)  serves  two 
purposes.  First,  the  function’s  transform  should  pass  the  desired  frequencies  F(w);  if 
substantial  aliasing  does  occur,  however,  it  could  be  potentially  beneficial  to  use  R(w)  to 
filter  out  the  aliased  portion  of  the  spectrum.  Second,  the  reconstruction  should  filter 
out  the  higher  order  replicas  of  F(w).  Remnants  of  the  high  frequency  replicas  represent 
coherent  changes  to  the  image  spectrum;  they  represent  a  degradation  in  the  fidelity  of 
the  replicated  signal.  In  practice,  many  of  the  image  artifacts  labeled  as  "aliasing"  result 
in  fact  from  the  presence  of  the  spurious  higher  frequencies  and  not  from  spectrum 
overlap;  these  artifacts  can  be  corrected  at  the  display. 


4.0  Display  of  Sampled  Imagery  Without 
Display  Processing 


In  many  cases,  the  eye  and  normal  display  blur  spots  do  not  provide  the  ideal 
reconstruction  described  in  Section  3.0;  they  do  not  provide  a  sharp  differentiation 
between  the  desired  original  spectrum  and  th'e  rfeplicas  at  each  sample  frequency.  A 
simplified  model  of  a  second  generation  thermal  imager  will  serve  as  an  example.  In 
Figure  5,  the  "original  spectrum"  represents  the  transform  of  the  analog  output  from  an 
F/3  narrow  field  of  view  sensor  using  the  Army  standard  focal  plane.  In  this  case,  the 
optical  blur  is  somewhat  larger  than  the  detector  and  sampling  rate  is  two  samples  per 
dwell.  The  sampled  signal  would  actually  be  the  input  scene  convolved  with  the  sensor 
MTF.  For  purposes  of  this  example,  we  will  use  the  MTF  itself;  consider  the  scene  to  be 
a  point  source  or  highly  random.  The  MTF  for  the  display  and  eye  are  based  on  a  6  inch 
by  8  inch,  flat  panel  display  with  50%  pixel  fill  factor  and  a  28  inch  eye  distance. 

This  example  sensor  is  well  sampled,  perhaps  over-sampled;  adjacent  replicas 
of  the  input  spectrum  do  not  overlap.  As  shown  in  Figure  5,  this  eye/display  combination 
causes  only  a  slight  drop  in  the  high  frequency  spatial  response  of  the  sensor  system;  i.e., 
the  high  end  of  the  original  spectrum  is  not  attenuated.  However,  the  eye/display 
combination  does  not  completely  "filter  out"  the  adjacent  replica  of  the  input  spectrum. 
As  illustrated  in  the  figure,  the  image  spectrum  has  significant  components  beyond  half 
the  sample  frequency.  TTus  high  frequency  signal  is  not  "noise;"  it  represents  a  change  in 
shape  or  intensity  pattern  of  the  image.  Although  significant  aliasing  is  not  present, 
sample  phase  artifacts  would  still  be  visible. 

If  the  sampling  artifacts  prove  to  be  bothersome,  one  corrective  action  is  to 
increase  the  sample  rate  and  improve  display  resolution.  The  increased  sample  rate 


would  further  separate  the  high  frequency  replicas  from  the  original  spectrum;  the 
eye/display  filtering  would  then  be  more  effective,  although  the  display  MTF  would 
increase  due  to  the  smaller  pixel  size.  Figure  6  illustrates  what  happens  when  the  sample 
rate  is  doubled;  the  replicate  spectra  are  more  widely  spaced,  with  the  result  that  the  eye 
provides  better  filtering  of  the  unwanted  high  frequencies.  The  spectrum  of  the  output 
is  a  closer  match  to  the  input  spectrum. 

Improved  image  quality  at  the  display  can  be  supported  without  increasing 
sensor  sample  rate.  Since  significant  aliasing  is  not  present  even  at  the  lower  sample  rate 
shown  in  Figure  5,  values  of  the  input  signal  at  the  mid-points  between  samples  can  be 
interpolated  from  that  data.  That  is,  we  find  an  r(x)  with  transform  F(w)  that  does  not 
degrade  the  desired  spectrum  but  will  filter  out  the  adjacent  replicate  spectra;  this  r(x)  is 
convolved  (mathematically,  in  a  computer  or  processor)  with  the  input  data  to  generate 
an  estimate  of  the  value  of  the  input  signal  at  points  intermediate  to  the  sample  points; 
the  expanded  set  of  samples,  original  data  plus  interpolated  values,  drive  the  higher 
resolution  display. 

K  sensor  imagery  is  just  sufficiently  sampled  to  avoid  significant  aliasing,  but 
not  excessively  sampled,  then  the  replicated  spectra  above  and  below  the  base-band 
(original)  spectrum  will  be  closely  spaced  to  the  original  but  not  overlapping.  In  order  to 
discriminate  the  original  spectrum  from  the  replicates,  we  would  like  an  R(w)  with  a  flat 
"passband"  to  frequency  w^/Z  and  a  sharp  cutoff  beyond. 


5.0  Reconstruction  Functions 


In  theory  a  perfect  r(x)  exists,  but  perfection  can  not  be  attained  in  a  realizable 
system.  However,  very  good  reconstruction  functions  can  be  easily  implemented.  In  this 
section,  the  Sampling  Theorem  will  be  described  and  its  practical  limitations  discussed; 
examples  will  then  be  given  of  practical  extensions  of  the  Sampling  Theorem. 

Referring  to  Figure  4  and  ^nation  2,  if  f(x)  is  band-limited  to  w^/Z  so  that  the 
replicas  of  F(w)  do  not  overlap,  and  if  R(w)  is  a  rectangular  function  centered  at  zero 
and  with  full  width  w^,  then  G(w)  will  equal  F(w)  and  f(x)  will  be  perfectly  replicated. 
Since  the  Fourier  Transform  of  a  rectangular  function  is  a  sample  function  [sin(x)/(x)], 
Equation  1  becomes: 


gW  =  I  f(nX)-sin(7tx/X-n7T)/(-nx/X-mr) 
all  n 


Equation  4 


The  Sampling  Theorem  states  that,  for  a  signal  f(x)  for  which  the  Fourier 
Transform  has  no  components  above  W5/2  inclusive,  the  function  can  be  entirely 
reconstructed  by  the  above  series  of  sample  functions.  That  is,  a  band-limited  function 
can  be  uniquely  determined  from  its  values  at  a  sequence  of  equidistant  points,  l/w^ 
apart.2  The  Sampling  Theorem  provides  the  capability  to  find  the  value  of  f(x)  at  points 
intermediate  to  the  sample  points. 

The  Sampling  Theorem  has  practical  limitations  in  that  realizable  signals  can 
not  be  ideally  sampled.  A  signal  can  not  be  both  band-limited  and  of  limited  extent;^  an 
ideal  reconstruction  would  require  either  infinite  sample  rate  or  gathering  an  infinitude 
of  samples.  Although  the  work  of  Landau  and  Poliak  a5  shows  that  the  sample  function 
expansion  is  not  theoretically  the  best  choice  for  realizable  signals,  in  practice  the 
Sampling  Theorem  does  hold  in  an  approximate  sense  and  is  a  useful  guide. 

The  Sampling  Theorem  is  awkward  to  apply  because  the  sample  function  falls 
off  slowly.  In  Equation  4,  if  we  are  interested  in  approximating  g(x)  at  a  point,  the 
contribution  to  the  approximation  from  a  distant  sample  falls  off  inversely  with  the 
distance  to  the  sample.  In  an  imaging  sensor,  samples  distant  by  more  than  a  spread 
function  diameter  are  not  contributing  real  information  content  to  the  approximation. 
However,  convolution  with  the  sample  functfon'can  not  be  arbitrarily  truncated  since 
contributions  from  distance  samples  tend  to  cancel  the  contributions  from  adjacent 
distant  samples.  Truncating  a  sine  wave  reconstruction  function  results  in  a  new 
reconstruction  function  with  a  transform  which  is  the  convolution  of  a  rectangular 
function  with  a  sine  wave.  The  new  R(w)  has  very  poor  reconstruction  properties. 

Reconstruction  functions  which  have  a  width  of  only  a  few  samples  and  retain 
the  desired  frequency  characteristics  can  be  generated  by  applying  a  smooth  window  to 
the  sample  function.  The  reconstruction  function  used  in  Figure  3  is  a  sample  function 
multiplied  by  a  Gaussian  which  falls  to  about  3%  amplitude  at  the  third  sample  and  the 
function  is  truncated  at  the  third  sample.  A  similar  technique  can  be  used  to  generate 
reconstruction  functions  of  different  widths;  a  wider  function  has  a  steeper  cutoff  at  the 
desired  band-limit. 

Figure  7  shows  the  reconstruction  filters  obtained  when  using  functions  with 
width  of  four,  six  and  eight  samples.  The  calculation  of  each  interpolation  point  would 
involve  a  series  with  four,  six  or  eight  terms,  respectively.  Also  shown  in  the  figure  is  the 
transform  of  the  triangular  wave  used  in  Figure  2.  Using  the  triangular  function  for 
reconstruction  provides  a  two  point  interpolation.  The  desirable  qualities  of  a 
reconstruction  function,  unit  amplitude  in  the  passband  and  a  sharp  cutoff,  are  enhanced 
by  using  more  points  in  the  reconstruction. 


6.0  Examples 


In  this  section,  examples  will  be  given  to  illustrate  that  image  quality  is 
enhanced  by  using  the  reconstruction  techniques  described  in  Section  5.0.  It  is  quite 
often  possible  to  significantly  enhance  displayed  image  quality  without  taking  additional 
sensor  samples. 

The  baseline  images,  referred  to  below  and  in  the  figures  as  "original," 
represent  good  quality,  black  and  white  video.  These  images  are  obtained  using  a  digital 
frame  grabber  and  a  high  quality.  Charge  Coupled  Device  camera.  The  Cohu  camera  is  a 
frame  transfer  device  with  760  horizontal  by  480  vertical  pixels.  The  camera  output  is 
digitized  with  a  Data  Translation  video  frame  grabber  taking  512  by  512  samples  with  8 
bit  accuracy.  The  digitizer  electronics  limits  the  video  bandwidth  to  4.5  MHz. 

The  original  image  at  the  top  left  of  Figure  8  shows  a  framed  picture  of  an 
AH-1  Cobra  Helicopter  hanging  on  the  laboratory  wall.  At  top  right  in  the  figure,  the 
helicopter  image  is  zoomed  eight  times  by  pixel  replication.  TTie  bottom,  right  picture  in 
Figure  8  is  generated  by  zooming  the  helicopter  image  eight  times  using  bi-linear 
interpolation.  Modified  Sample  Theorem  interpolation  was  used  to  process  the  bottom, 
left-hand  picture.  Sample  Theorem  reconstruction  clearly  provides  the  best  image 
quality. 


In  Figure  9,  the  original  image  is  down-sampled  to  256  by  256  pixels.  Three 
out  of  four  of  the  original  samples  are  discarded  in  order  to  represent  a  sensor  which 
takes  one  forth  the  samples.  The  top  right  picture  shows  the  result  of  replicating  the 
down-sampled  image  sixteen  times.  The  bottom,  right  picture  results  from  bi-linear 
interpolation  and  the  bottom,  left  picture  from  Sample  TTieorem  reconstruction.  The 
images  at  the  bottom  of  Figure  9  look  like  Cobra  Helicopters  even  though  the 
reconstruction  uses  only  a  quarter  of  the  sensor  samples  compared  to  Figure  8.  If  the 
top,  right  picture  in  Figure  9  is  used  as  the  basis  of  comparison,  the  improved  image 
quality  represented  by  the  bottom,  left  picture  in  Figure  9  is  obtained  solely  through  the 
use  of  display  reconstruction  techniques,  whereas  four  times  as  many  sensor  samples  are 
required  to  achieve  the  improved  image  quality  represented  by  the  pictures  in  Figure  8. 

The  original  image  of  a  face  is  shown  at  the  top,  left-hand  side  in  Figure  10. 

To  the  right  in  the  figure,  this  face  is  down-sampled  to  256  by  256  pbcels.  The  bottom, 
left  picture  results  from  Sample  Theorem  reconstruction.  Only  minor  differences  can  be 
discerned.  / 


The  top,  right  image  in  Figure  11  is  obtained  by  four  pbcel  replications  of  the 
original  face  shown  at  the  top,  left  of  Figure  10.  The  top,  left  picture  in  Figure  11  results 
from  eight  pixel  replications  of  the  down-sampled  image  shown  at  the  top,  right  of 
Figure  10.  The  bottom  image  in  Figure  11  is  a  Sample  Theorem  reconstruction  from  the 
down-sampled  data.  The  picture  constructed  from  fewer  samples  using  good 
reconstruction  techniques  is  a  close  match  to  the  pbcel  replicated  image  which  is 
generated  from  four  times  the  number  of  samples. 


7.0  Conclusion 


In  many  practical  applications,  poor  image  quality  or  sample  artifacts  result 
from  poor  signal  reconstruction  rather  than  from  inadequate  sampling.  Good  replication 
of  a  sampled  image  requires  that  sample  spacing  be  small  enough  that  aliasing  does  not 
occur.  However,  once  a  minimum  sample  rate  has  been  reached  for  a  given  sensor, 
improved  display  techniques  can  substitute  for  increased  sampling  and  perhaps  save 
system  cost  and  complexity. 

The  examples  illustrate  that  image  quality  can  be  improved  by  display 
processing.  Further,  near  ideal  reconstruction  functions  exist  which  are  only  a  few 
samples  wide.  Improved  image  reconstruction  by  interpolating  the  input  data  is  a 
practical  alternative  to  taking  additional  sensor  samples. 
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Figure  1.  Reconstruction  of  a  Signal  using  a 
Rectangular  Function  to  Replicate  the  Samples. 


Figure  2.  Reconstruction  of  a  Signal  using  a 
Triangular  Function  to  Bi-linear  Interpolate. 


Figure  3.  Reconstruction  of  a  SignaUusing  a 
Gaussian  Windowed  Sine  Function 


Figure  4.  Notional  Plot  of  F(w)  Replicated  at  Sample 
Frequencies  and  'Filtered’  by  R(w). 


Figure  5.  Spectra  Associated  with  the  Sampled  Output  of 
the  Hypothesized  Thermal  Imaginq  system. 


Figure  6.  Spectra  Associated  with  Doubling  the  Sample 
Rate  in  Figure  5  Above. 


RECONSTRUCTION  FILTER  AMPLITUDE  RESPONSE 


*  u 


SPATIAL  FREQUENCY  (Sample  Freq  -  2) 
Figure  7.  Example  R(w)  for  Functions  of  Limited  Width. 
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Figure  10. 


USING  SAMPLE  THEOREM 
TO  ZOOM  (8)  TIMES 


